The Power of Experiments: Decision Making in a Data-Driven World
- Downloads：5348
- Type：Epub+TxT+PDF+Mobi
- Create Date：2022-10-26 05:51:50
- Update Date：2025-09-06
- Status：finish
- Author：Michael Luca
- ISBN：0262043874
- Environment：PC/Android/iPhone/iPad/Kindle
- Download
Summary
How organizations--including Google， StubHub， Airbnb， and Facebook--learn from experiments in a data-driven world。
Have you logged into Facebook recently？ Searched for something on Google？ Chosen a movie on Netflix？ If so， you've probably been an unwitting participant in a variety of experiments--also known as randomized controlled trials--designed to test the impact of different online experiences。 Once an esoteric tool for academic research， the randomized controlled trial has gone mainstream。 No tech company worth its salt （or its share price） would dare make major changes to its platform without first running experiments to understand how they would influence user behavior。 In this book， Michael Luca and Max Bazerman explain the importance of experiments for decision making in a data-driven world。
Luca and Bazerman describe the central role experiments play in the tech sector， drawing lessons and best practices from the experiences of such companies as StubHub， Alibaba， and Uber。 Successful experiments can save companies money--eBay， for example， discovered how to cut $50 million from its yearly advertising budget--or bring to light something previously ignored， as when Airbnb was forced to confront rampant discrimination by its hosts。 Moving beyond tech， Luca and Bazerman consider experimenting for the social good--different ways that govenments are using experiments to influence or "nudge" behavior ranging from voter apathy to school absenteeism。 Experiments， they argue， are part of any leader's toolkit。 With this book， readers can become part of "the experimental revolution。"
Have you logged into Facebook recently？ Searched for something on Google？ Chosen a movie on Netflix？ If so， you've probably been an unwitting participant in a variety of experiments--also known as randomized controlled trials--designed to test the impact of different online experiences。 Once an esoteric tool for academic research， the randomized controlled trial has gone mainstream。 No tech company worth its salt （or its share price） would dare make major changes to its platform without first running experiments to understand how they would influence user behavior。 In this book， Michael Luca and Max Bazerman explain the importance of experiments for decision making in a data-driven world。
Luca and Bazerman describe the central role experiments play in the tech sector， drawing lessons and best practices from the experiences of such companies as StubHub， Alibaba， and Uber。 Successful experiments can save companies money--eBay， for example， discovered how to cut $50 million from its yearly advertising budget--or bring to light something previously ignored， as when Airbnb was forced to confront rampant discrimination by its hosts。 Moving beyond tech， Luca and Bazerman consider experimenting for the social good--different ways that govenments are using experiments to influence or "nudge" behavior ranging from voter apathy to school absenteeism。 Experiments， they argue， are part of any leader's toolkit。 With this book， readers can become part of "the experimental revolution。"
Reviews
Bianca
，
The book is a good way to get into the topic of experiments and nice to read if you expect easy and plain reading， but it did not meet my expectations as I was hoping for more than a list of companies having done experiments。
J
，
Great book！ This book gives a great overview of how experiments can be applied in various sectors （government， sciences， tech， education， etc。） and what kind of decisions can be derived from experimentations。I found the examples of experimentation from a historical standpoint to modern-day applications very interesting to learn about。 This book is easy to read for anyone interested in how experiments are applied today to improve apps or the status quo。 It gives a good general overview of experim Great book！ This book gives a great overview of how experiments can be applied in various sectors （government， sciences， tech， education， etc。） and what kind of decisions can be derived from experimentations。I found the examples of experimentation from a historical standpoint to modern-day applications very interesting to learn about。 This book is easy to read for anyone interested in how experiments are applied today to improve apps or the status quo。 It gives a good general overview of experimentation but it does not go into depth about the statistics behind running experiments。 I love learning about behavioral economics and I'm early in my career in data science。 Overall， a great read for an intro to experimentation。 。。。more
davexfaulk
，
"here is a company that did an experiment。 and here is another。 and here is a nonprofit experiment。 and a tech company experiment。"That's basically it。 I had high expectations for this book because experimentation can be so interesting and deep， but this felt incredibly flaccid。 "here is a company that did an experiment。 and here is another。 and here is a nonprofit experiment。 and a tech company experiment。"That's basically it。 I had high expectations for this book because experimentation can be so interesting and deep， but this felt incredibly flaccid。 。。。more
Jacob
，
I thought this was pretty good， and I’m surprised that the reviews are lower！ It covers the history of experimentation， its growing popularity， and some interesting examples in the tech industry， as well as behavior change in the public sector （think tax collection， education， health， etc。）。 The examples from the tech industry were the most relevant for me and I especially liked the one about eBay realizing that paying for Google ads on the “eBay” term is a waste of money， since people who type I thought this was pretty good， and I’m surprised that the reviews are lower！ It covers the history of experimentation， its growing popularity， and some interesting examples in the tech industry， as well as behavior change in the public sector （think tax collection， education， health， etc。）。 The examples from the tech industry were the most relevant for me and I especially liked the one about eBay realizing that paying for Google ads on the “eBay” term is a waste of money， since people who type that are high-intent and will go to ebay。com either way， whereas advertising on terms like “used guitar” is more beneficial。 I will say， the book was very high-level and definitely written for managers more than for practitioners — I don’t think p-values were mentioned once — but the authors made this clear from the beginning。 There was a fair amount of information there that I’ve read in other books but that’s because I’ve read more books on experimentation than any sane person would。 Can’t blame the authors for that。 。。。more
Anne
，
It does prove the title and is nice to read， but mainly gives a lot of examples about experiments and only a very general overview of what a good experiment is。 It would have been more valuable if they would have structured it more and dived deeper into the experiments part。 I really do recommend reading it if you’re not completely sure as to why its important to conduct experiments。 Some examples are a real eye opener to test even your most basic and seemingly obvious assumptions。
Walter
，
This book is a love letter to "Nudge" by Richard Thaler。 Seriously。It is laden with references to the book as well as related studies in the field of behavioral economics。 Matter of fact， if you do a shot every time you read "nudge" on the page， you'd most likely die of cirrhosis before you get to page 150。。。 I was expecting more of a "deep-dive" per se， but most of the time is spent recounting material from other texts and adding some context。 It's not bad， but it's not special。The good:They wi This book is a love letter to "Nudge" by Richard Thaler。 Seriously。It is laden with references to the book as well as related studies in the field of behavioral economics。 Matter of fact， if you do a shot every time you read "nudge" on the page， you'd most likely die of cirrhosis before you get to page 150。。。 I was expecting more of a "deep-dive" per se， but most of the time is spent recounting material from other texts and adding some context。 It's not bad， but it's not special。The good:They wisely chose to spend some time on the way industry giants like Uber， Facebook， AirBnB， and others employ experiments to drive decision-making。 The bad:It's all a bit shallow， like when your friends tell you to come over to have a splash in the pool but it is a kitty pool。。。 They gloss over practical terms for the sake of keeping things friendly for a presumably non-technical crowd。I can't recommend it if you have read "Thinking Fast and Slow"， "The Undoing Project"， or any of the other behavioral economics classics。 It's mostly rehashed。 However， if you are completely new to experiments （or the idea of using experiments to drive managerial decisions） then this book will prove a fast and enjoyable read。 。。。more
Daan Dumon
，
Very boring and repeats the same point for different media。 Great book to fall asleep too though。
George Mount
，
Ended up being an enjoyable read， but wasn't quite what I was expecting。 It felt very academic with lots about the rise of experimental economics， econ vs psychology， etc。 Most readers would have likely benefited from more specific tactics for deploying experiments in their companies。 Ended up being an enjoyable read， but wasn't quite what I was expecting。 It felt very academic with lots about the rise of experimental economics， econ vs psychology， etc。 Most readers would have likely benefited from more specific tactics for deploying experiments in their companies。 。。。more
Jordan Keyes
，
My manager， despite being in a very controlled， back-end role， loves experiments。 Or， at least the idea of them。 In an effort to push his team （and other teams within the business） to challenge the status quo a bit， he assigned The Power of Experiments， by Harvard Business School professors Michael Luca & Max Bazerman， as a group summer read。 I wouldn’t exactly call it a fun beach read， but this book had some interesting tidbits of information。 Largely though， it was a typical business book。 As My manager， despite being in a very controlled， back-end role， loves experiments。 Or， at least the idea of them。 In an effort to push his team （and other teams within the business） to challenge the status quo a bit， he assigned The Power of Experiments， by Harvard Business School professors Michael Luca & Max Bazerman， as a group summer read。 I wouldn’t exactly call it a fun beach read， but this book had some interesting tidbits of information。 Largely though， it was a typical business book。 As part of this summer book club， I took notes throughout my read to prepare for our Teams discussions。 There were three main sections in the book。 The first served as a primer of how experiments got started within psychology and economics， and later how they rose to prominence in policy making。 One of the examples they gave was a simple experiment the UK tax authority designed to have debtors pay on time more often。 Through some simple trials of changing words around in the tax letters， they found a way to increase people paying back their taxes。 It was a simple fix and the British government received millions more back in taxes。 The second and most lengthy part of the book dealt with examples of experiments within the tech sector。 Luca and Bazerman basically just summarize problems that some tech companies have recently faced， and how they used an experimental mindset to gain greater insight。 It includes several instances of how experiments can lead a company awry。 But it mostly includes examples of nuanced successes。 An example of this is when Alibaba allowed sellers to give coupons to customers who left items in their cart for over 24 hours。 This helped the specific item sell but didn’t lead to long-term profit。 In fact， it’s emphasized in this part of the book that experiments are often better at focusing on short-term optimization/profits while missing long-term risks （which are harder to measure）。 Experimenting for the Social Good is the third and final section of The Power of Experiments。 It contains， in my opinion， the most interesting chapter。 This chapter centers on behavioral experiments within the political sphere。 I think this is comparatively more thought-provoking because it finally didn’t revolve around profit。 In the 90s， some Yale political scientists asked - what motivates people to vote？ They found out that door to door canvassing is by far the most effective method， while postcards had only a marginal effect， and phone calls didn’t help at all。 Which leads me to think， why are the latter two methods still being used （to annoying effect） today at all？Similarly， a separate political behavioral economist， Todd Rogers， looked at how to get people to vote。 He utilized a concept called “nudging”， which prompts people to make concrete plans， and thus increases the likelihood of actually completing a goal （in this case， making a plan to vote）。 His experiments show that focusing your attention on the individuals who will vote is better than wasting time on those who were never planning on voting anyway。Additionally， I wrote down some particularly valuable points related to experimentation。 Here are a few: lab experiments can help us know whether and when an effect might be relevant， but won’t predict the exact effect in any particular real-world setting; company leaders are central to creating an organization dedicated to experimentation and evidence; short-term outcomes can be misleading （and it can be valuable to invest in tracking data for a longer period）; when deciding whether an experiment is ethical， it’s important to weigh long-term benefits against short-term opportunity costs; and finally， evidence trumps intuition。 The most striking quote I read was by Google’s chief economist， Hal Varian。 He said， “Why debate this point， when we can simply run an experiment to find out？” Getting corporations to move to that mindset is very valuable now and will continue to be critical in the future。Another side point I’ll mention is that the humor in this book is non-existent。 Occasionally Luca and Bazerman will try to throw a joke in， but they consistently left me shaking my head。 It was among the driest reads I’ve had in a while。Overall， business books like this one can be tough。 They often have good intentions and can offer solid advice。 They can help you learn more about yourself and how to improve in the workplace。 However， I would also say most full-length business books （and self-help books too， for that matter） can be summarized into just a few pages worth of quality information。 The Power of Experiments fits this mold too， unfortunately。 It mostly focuses on experimentation examples that are really only relevant to those who work in the marketing or analytics functions of large tech companies （which the vast majority of people don’t）。There are still some good ideas here that are worth talking about。 It got my team to discuss how we can improve our processes and how we approach feedback from others。 And， of course， how we can further experiment to create better change for the business。 3 / 5。 。。。more
nawaf
，
My experience I liked the wide spectrum of examples covered by the writer。He gradually get you into the subject。My first ever English book。 Very good start for my readings。
Chelsea
，
Solid introduction to the "why" of social experiments， including business。 Not really about the "how。" Had a lot of very familiar examples that are brought up in this realm a lot （Have you heard about eBay's advertising experiment？？？） but I thought it was decently-written， a rarity in business writing。 It was heavily focused on behavioral economic experiments， due to the popularity and the authors' background， I think， but I appreciated the point that these biases are often plucked out of their Solid introduction to the "why" of social experiments， including business。 Not really about the "how。" Had a lot of very familiar examples that are brought up in this realm a lot （Have you heard about eBay's advertising experiment？？？） but I thought it was decently-written， a rarity in business writing。 It was heavily focused on behavioral economic experiments， due to the popularity and the authors' background， I think， but I appreciated the point that these biases are often plucked out of their experimental context and thought of as universally applicable。 The chapter on AirBnb also was an interesting window into experimental resistance and ethics， but the authors did have a clear bone to pick on this。 Overall wish there had been more diversity of examples。 。。。more
Tran Ngan
，
The book is fine enough。 It was more helpful when I apply this 'experiment' attitude mentioned in my real life。 How 'experiments' help me understand more about people around me， the choices I made and how can I make better decisions in the future :） The book is fine enough。 It was more helpful when I apply this 'experiment' attitude mentioned in my real life。 How 'experiments' help me understand more about people around me， the choices I made and how can I make better decisions in the future :） 。。。more
Christian
，
This is a good intro to experimentation in tech。 The problem is that most people in tech are well-versed in experimentation。 They explain companies like Google， and older ones like Yahoo！ （？） to illustrate things we’ve known for a decade。 The book isn’t bad， it’s just woefully late。
Nilesh Makan
，
I've been interested in behavioural economics of late。 This book is a great summary of the discipline of behavioural economics， and how this can be applied in business。 The focus is predominantly on the tech sector， mostly because it is this sector that， though technology， can run more experiments and collect the most amount of data through those experiments to test a hypothesis。 I thoroughly liked the way the book promoted that businesses should use experiments to aide decision making， but not I've been interested in behavioural economics of late。 This book is a great summary of the discipline of behavioural economics， and how this can be applied in business。 The focus is predominantly on the tech sector， mostly because it is this sector that， though technology， can run more experiments and collect the most amount of data through those experiments to test a hypothesis。 I thoroughly liked the way the book promoted that businesses should use experiments to aide decision making， but not be over reliant on the experiment。 I also appreciate that most experiments are there to drive short term decisions， and how we should design experiments to have longevity， so that we create sustainable businesses。Behavioural economics is a powerful tool， and with great power comes great responsibility。 As with all powerful tools and technologies， we must use this to build better， more inclusive societies， to use these tools too improve humanity， and not for nefarious purposes， or that for increased profit or greed。 I thoroughly enjoyed the book， as it is packed with interesting examples of how experiments were run， and provides insights around how you can create an organisation that experiments。 。。。more
Phil Dearson
，
It’s a gentle introduction to the benefits of evidence-based decision-making。 Has a few high profile case studies。 You’re not going to get any depth about decision science or behavioural economics。
Chris Boutté
，
Absolutely amazing book。 This book does a great job teaching you how to use experiments in your everyday life as well as at work to test out different methods。 It’s also an Important book to help us learn which studies we should pay attention to or ignore in media
Aidan Gibson
，
As dry as an academic book， without the rigor of one。 There is practically nothing in this book that hasn't already been said by Eric Ries and Daniel Kahneman YEARS ago。I have no idea who they were trying to appeal to here， it's too shallow to be an intellectual book and the writing is too poor to be a proper popsci book。 If you have nothing to say， why publish？ As dry as an academic book， without the rigor of one。 There is practically nothing in this book that hasn't already been said by Eric Ries and Daniel Kahneman YEARS ago。I have no idea who they were trying to appeal to here， it's too shallow to be an intellectual book and the writing is too poor to be a proper popsci book。 If you have nothing to say， why publish？ 。。。more
Wen Rei
，
One of the chapters that discuss the experimentations carried out by tech companies is eye-opening as companies could not only generate more revenue when they learnt about consumers' behaviour， but also re-evaluate current decisions to save lots of money potentially。 Real-world examples such as Facebook， Google， Airbnb and so on were brought out and it justifies the author's point that companies are experimenting heavily and more companies will follow suit。 One of the chapters that discuss the experimentations carried out by tech companies is eye-opening as companies could not only generate more revenue when they learnt about consumers' behaviour， but also re-evaluate current decisions to save lots of money potentially。 Real-world examples such as Facebook， Google， Airbnb and so on were brought out and it justifies the author's point that companies are experimenting heavily and more companies will follow suit。 。。。more
Todd Cheng
，
The book covers key experimentation done in business and industry and how it can impact the goals。 Experimenting is a complement to intuition with data and tests。 Often our gut reaction or what seem intuitive does not result in the best solution。 Testing helps meet key purposes like verifying theory， understanding magnitudes， evaluating policy or products， or exploring fact finding。 it is best to use a series of experimentation to create a better frameworks。 Similar to how a multimodel frame wor The book covers key experimentation done in business and industry and how it can impact the goals。 Experimenting is a complement to intuition with data and tests。 Often our gut reaction or what seem intuitive does not result in the best solution。 Testing helps meet key purposes like verifying theory， understanding magnitudes， evaluating policy or products， or exploring fact finding。 it is best to use a series of experimentation to create a better frameworks。 Similar to how a multimodel frame work provide a better understanding a series of experimentation leads to a better framework。 Test also focus on narrow or short term outcomes。 Often the intuition is wrong and experimentation helps detect the impacts on harder longer term goals。 The book covers how this is being done in many business and government。 It also deletes lightly into the evolving ethics of this in the edge of digital everything with Uber， Facebook， and university。 。。。more
Bookers
，
Rating-🌟🌟🌟。5Pages-211How many experiments do you think you’ve participated in over the past year？ We’re talking about randomized controlled trials—experiments designed to test the impact of different treatments by randomly assigning you and other participants （often called subjects） to various treatment conditions， like those you might have participated in if you took Psych 101 or if you’ve tried out an experimental drug。 So， what’s your number？ At first blush， you might think the answer is zero Rating-🌟🌟🌟。5Pages-211How many experiments do you think you’ve participated in over the past year？ We’re talking about randomized controlled trials—experiments designed to test the impact of different treatments by randomly assigning you and other participants （often called subjects） to various treatment conditions， like those you might have participated in if you took Psych 101 or if you’ve tried out an experimental drug。 So， what’s your number？ At first blush， you might think the answer is zero。 But unless you live in a bunker with no Internet access， you’ve likely participated in many experiments over the past year。 If you logged onto Facebook right now， there’s a good chance you’d be an unwitting subject in a variety of the company’s ongoing experiments as you scroll through your News Feed and peruse the ads being shown to you。 You are also likely to be a test subject if you search for an item on Google， watch a movie on Netflix， respond to email surveys， or call companies for customer support。A book backed up with facts and figures and experiments ofcourse。📚Gives the examples of successful and unsuccessful experiments of the of tech giants like #Google #Uber #Facebook and #airbnbBy the end of the book you will be realising that once you start looking， you can’t help noticing incidental experiments everywhere。 Do you see any in your organization？ If so， what insights might you glean from them？ Give this a read。https://amzn。to/3d5lXY0 。。。more
Greg Stoll
，
This book was fine enough。 It's all about how companies and governments should do experiments rather than using their intuition about what they think will work。 There's some amount of useful stuff if you're actually in a position to do such things， but other than that the book is mostly examples of places that have done experiments。 A few interesting points:- The British government did an experiment on the wording they used on letters to people that hadn't paid their taxes。 Turns out the most ef This book was fine enough。 It's all about how companies and governments should do experiments rather than using their intuition about what they think will work。 There's some amount of useful stuff if you're actually in a position to do such things， but other than that the book is mostly examples of places that have done experiments。 A few interesting points:- The British government did an experiment on the wording they used on letters to people that hadn't paid their taxes。 Turns out the most effective wording to get people to pay their taxes is to include something like "9 out of 10 people in the UK pay their taxes on time。 You are in the very small minority of people who haven't yet。" The effect is kinda small （the number of people who pay goes up from 35% to 37%）， but of course in aggregate that adds up to a lot of money！- For whether people want to be an organ donor， if it's an opt-in system 4-28% will sign up， but in an opt-out system 86-100%， so the default has a huge effect！ But a lot of people think it's possibly unethical to sign people up by default。 A thought was to move to "active choice"， where the state explicitly asks people whether they want to be organ donors （so there's no default）， but after doing an experiment fewer people signed up than even in the opt-in system！- There's a whole chapter about racial bias on Airbnb， starting with the story of an African-American man trying to book a room but being repeatedly rejected， probably because of his race。 The authors ran an experiment sending a bunch of rental inquiries that were identical except for the name - half were from （fake） guests with first names common among white people （"Brett"， 'Todd"） and half were from fake guests with first names common among black people （"Darnell"， "Jamal"）。 No photos were included for simplicity。 I'm guessing at this point you will not be shocked to read that the names common among black people got 16% fewer yeses from hosts。 （a similar study has been done with resumes for jobs （here's the original PDF）， and the results were similar but the effect was 3 times as large！） Once this study was published， Airbnb was pretty embarrassed and ran some experiments to try to reduce discrimination， but refused to say what experiments they had run or what the results were。 。。。more
Duygu
，
Luca and Bazerman took a subject which has great potential and wasted it with presenting the reader excerpts from Thaler's Nudge and Duckworth's Grit， and talking about some of the research centers the writers worked for。 The rest of the book is a few examples from tech industry。 The message this book is trying to deliver is "You can use tests/experiments to answer your questions" which unfortunately for a book written in 2020 is at least 15 years too late。 I admit that I am not an average reade Luca and Bazerman took a subject which has great potential and wasted it with presenting the reader excerpts from Thaler's Nudge and Duckworth's Grit， and talking about some of the research centers the writers worked for。 The rest of the book is a few examples from tech industry。 The message this book is trying to deliver is "You can use tests/experiments to answer your questions" which unfortunately for a book written in 2020 is at least 15 years too late。 I admit that I am not an average reader。 I spent two years in graduate school， studying behavorial economics and experiments。 I design and run experiments in non-tech organizations as a part of my job。 I know the content of this book by hearth。 What the authors don't realize is， every single organization I've worked for or interviewed with since 2008， every single one of them knew that they could use an experiment to answer a question。 What they didn't know or still don't know is， self selection bias， the concept of statistical significance， controlling for confounder effects or controlling for interactions。 Most of these companies （including tech） run multiple tests on the same subject at the same time and assume independent results， just show why it isn't。This book gave me absolutely nothing and it will not give anything new to a reader who at least knows what an A/B Test is。 I have wasted $16 and 2ish hours on this。 Learn from my mistake and don't waste yours。 。。。more
Nitin
，
The book is a great introduction to experiments and experimental thinking。 The examples given in the book are really eye-opening and the discussions on them are also insighful。 I didn't care for the structure of the book but the content was really amazing。 The book is a great introduction to experiments and experimental thinking。 The examples given in the book are really eye-opening and the discussions on them are also insighful。 I didn't care for the structure of the book but the content was really amazing。 。。。more
Andreas Aristidou
，
This is a thoughtful， short and concise book about an in-demand topic， written by leading academics in the field of behavioral economics and experimentation。 Luca and Bazerman provide examples to illustrate their points from various angles without making the book simply a collection of bullet points。 They dive into the historical roots of behavioral science and experiments， focusing on the fields of Psychology and Economics。 The authors cleverly describe the technicalities of experiments and beh This is a thoughtful， short and concise book about an in-demand topic， written by leading academics in the field of behavioral economics and experimentation。 Luca and Bazerman provide examples to illustrate their points from various angles without making the book simply a collection of bullet points。 They dive into the historical roots of behavioral science and experiments， focusing on the fields of Psychology and Economics。 The authors cleverly describe the technicalities of experiments and behavioral insights with easy distinctions between similar and often confusing concepts， precluding the need for more technical detail。 I particularly enjoyed the structured and organized dive into tech industry experiments。 Each chapter of Part II describes one such experiment / tech company， accompanied by a lesson that illustrates the “take-away” point。 Occasionally， I found their background stories to be somewhat myopically focused on researchers and organizations affiliated with their institution – Harvard University。 As someone who is very familiar with the literatures they touch upon， I would expect to see a slightly broader perspective of how the contributions of other institutions helped shape the state of behavioral experimentation。 Overall however， I would highly recommend this book to anyone – not necessarily to people interested in experiments。 It’s such an important， widespread and on the rise topic， presented in a very accessible and illustrative manner， that it’s probably worth anyone’s time。 Read a detailed chapter-by-chapter summary on my blog here: https://aristidouandreas。com/book-rev。。。 。。。more
Theodore Kinni
，